Replica trick with real replicas: A way to build in thermodynamic homogeneity

We use real replicas to investigate stability of thermodynamic homogeneity of the free energy of the Sherrington-Kirkpatrick (SK) model of spin glasses. Within the replica trick with the replica symmetric ansatz we show that the averaged free energy at low temperatures is not thermodynamically homogeneous. The demand of minimization of the inhomogeneity of thermodynamic potentials leads in a natural way to the hierarchical solution of the Parisi type. Conditions for the global thermodynamic homogeneity are derived and evaluated for the SK and $p$-spin infinite range models.Comment: 6 pages, presented at SPDSA2004 Hayama (Japan), to appear in Progr. Theor. Phy

Computing a Knot Invariant as a Constraint Satisfaction Problem

We point out the connection between mathematical knot theory and spin glass/search problem. In particular, we present a statistical mechanical formulation of the problem of computing a knot invariant; p-colorability problem, which provides an algorithm to find the solution. The method also allows one to get some deeper insight into the structural complexity of knots, which is expected to be related with the landscape structure of constraint satisfaction problem.Comment: 6 pages, 3 figures, submitted to short note in Journal of Physical Society of Japa

Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model

Understanding the reasons for the success of deep neural networks trained using stochastic gradient-based methods is a key open problem for the nascent theory of deep learning. The types of data where these networks are most successful, such as images or sequences of speech, are characterized by intricate correlations. Yet, most theoretical work on neural networks does not explicitly model training data or assumes that elements of each data sample are drawn independently from some factorized probability distribution. These approaches are, thus, by construction blind to the correlation structure of real-world datasets and their impact on learning in neural networks. Here, we introduce a generative model for structured datasets that we call the hidden manifold model. The idea is to construct high-dimensional inputs that lie on a lower-dimensional manifold, with labels that depend only on their position within this manifold, akin to a single-layer decoder or generator in a generative adversarial network. We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a "Gaussian equivalence property"(GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times. This approach permits us to analyze in detail how a neural network learns functions of increasing complexity during training, how its performance depends on its size, and how it is impacted by parameters such as the learning rate or the dimension of the hidden manifold

Compressed sensing with l0-norm: statistical physics analysis and algorithms for signal recovery

Noiseless compressive sensing is a protocol that enables undersampling and later recovery of a signal without loss of information. This compression is possible because the signal is usually sufficiently sparse in a given basis. Currently, the algorithm offering the best tradeoff between compression rate, robustness, and speed for compressive sensing is the LASSO (l1-norm bias) algorithm. However, many studies have pointed out the possibility that the implementation of lp-norms biases, with p smaller than one, could give better performance while sacrificing convexity. In this work, we focus specifically on the extreme case of the l0-based reconstruction, a task that is complicated by the discontinuity of the loss. In the first part of the paper, we describe via statistical physics methods, and in particular the replica method, how the solutions to this optimization problem are arranged in a clustered structure. We observe two distinct regimes: one at low compression rate where the signal can be recovered exactly, and one at high compression rate where the signal cannot be recovered accurately. In the second part, we present two message-passing algorithms based on our first results for the l0-norm optimization problem. The proposed algorithms are able to recover the signal at compression rates higher than the ones achieved by LASSO while being computationally efficient

Ground-state configuration space heterogeneity of random finite-connectivity spin glasses and random constraint satisfaction problems

We demonstrate through two case studies, one on the p-spin interaction model and the other on the random K-satisfiability problem, that a heterogeneity transition occurs to the ground-state configuration space of a random finite-connectivity spin glass system at certain critical value of the constraint density. At the transition point, exponentially many configuration communities emerge from the ground-state configuration space, making the entropy density s(q) of configuration-pairs a non-concave function of configuration-pair overlap q. Each configuration community is a collection of relatively similar configurations and it forms a stable thermodynamic phase in the presence of a suitable external field. We calculate s(q) by the replica-symmetric and the first-step replica-symmetry-broken cavity methods, and show by simulations that the configuration space heterogeneity leads to dynamical heterogeneity of particle diffusion processes because of the entropic trapping effect of configuration communities. This work clarifies the fine structure of the ground-state configuration space of random spin glass models, it also sheds light on the glassy behavior of hard-sphere colloidal systems at relatively high particle volume fraction.Comment: 26 pages, 9 figures, submitted to Journal of Statistical Mechanic

Solution space heterogeneity of the random K-satisfiability problem: Theory and simulations

The random K-satisfiability (K-SAT) problem is an important problem for studying typical-case complexity of NP-complete combinatorial satisfaction; it is also a representative model of finite-connectivity spin-glasses. In this paper we review our recent efforts on the solution space fine structures of the random K-SAT problem. A heterogeneity transition is predicted to occur in the solution space as the constraint density alpha reaches a critical value alpha_cm. This transition marks the emergency of exponentially many solution communities in the solution space. After the heterogeneity transition the solution space is still ergodic until alpha reaches a larger threshold value alpha_d, at which the solution communities disconnect from each other to become different solution clusters (ergodicity-breaking). The existence of solution communities in the solution space is confirmed by numerical simulations of solution space random walking, and the effect of solution space heterogeneity on a stochastic local search algorithm SEQSAT, which performs a random walk of single-spin flips, is investigated. The relevance of this work to glassy dynamics studies is briefly mentioned.Comment: 11 pages, 4 figures. Final version as will appear in Journal of Physics: Conference Series (Proceedings of the International Workshop on Statistical-Mechanical Informatics, March 7-10, 2010, Kyoto, Japan

Machine learning and the physical sciences

Machine learning encompasses a broad range of algorithms and modeling tools used for a vast array of data processing tasks, which has entered most scientific disciplines in recent years. We review in a selective way the recent research on the interface between machine learning and physical sciences. This includes conceptual developments in machine learning (ML) motivated by physical insights, applications of machine learning techniques to several domains in physics, and cross-fertilization between the two fields. After giving basic notion of machine learning methods and principles, we describe examples of how statistical physics is used to understand methods in ML. We then move to describe applications of ML methods in particle physics and cosmology, quantum many body physics, quantum computing, and chemical and material physics. We also highlight research and development into novel computing architectures aimed at accelerating ML. In each of the sections we describe recent successes as well as domain-specific methodology and challenges

Marvels and Pitfalls of the Langevin Algorithm in Noisy High-Dimensional Inference

Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference. In this work, we carry out an analytic study of the performance of the algorithm most commonly considered in physics, the Langevin algorithm, in the context of noisy high-dimensional inference. We employ the Langevin algorithm to sample the posterior probability measure for the spiked mixed matrix-tensor model. The typical behavior of this algorithm is described by a system of integrodifferential equations that we call the Langevin state evolution, whose solution is compared with the one of the state evolution of approximate message passing (AMP). Our results show that, remarkably, the algorithmic threshold of the Langevin algorithm is suboptimal with respect to the one given by AMP. This phenomenon is due to the residual glassiness present in that region of parameters. We also present a simple heuristic expression of the transition line, which appears to be in agreement with the numerical results

On the cavity method for decimated random constraint satisfaction problems and the analysis of belief propagation guided decimation algorithms

We introduce a version of the cavity method for diluted mean-field spin models that allows the computation of thermodynamic quantities similar to the Franz-Parisi quenched potential in sparse random graph models. This method is developed in the particular case of partially decimated random constraint satisfaction problems. This allows to develop a theoretical understanding of a class of algorithms for solving constraint satisfaction problems, in which elementary degrees of freedom are sequentially assigned according to the results of a message passing procedure (belief-propagation). We confront this theoretical analysis to the results of extensive numerical simulations.Comment: 32 pages, 24 figure